One of the reasons I’ve been such a proponent of people at my current job reading Getting Real is that it describes the way I’ve approached development ever since I got started in the field. Now, mind you, I’m not a “classically trained” developer. As such, I just went with the way of doing things that made the most sense. Silly me. While the number of programmers doing things in a manner that makes sense is growing every day, we, members of the so-called agile development crowd, are still decidedly counterculture in the corporate world.

Anyway, some events occurred this week that made me feel a retrospective of one such development project was in order.

The nature of the corporate development beast

Now, I know that much of this won’t be news to developers who’ve been around a while, but just humor me. There may be some young, innocent souls among you who will benefit from hearing this now. ;)

You see, there are many companies where it’s assumed that programmers should never get involved in a project until a Business Analyst sits down with the project’s major stakeholders and asks them to sign off to a full list of functional requirements. In severe cases of overplanning, instead of just describing what the user wants the app to accomplish, they’ll start laying out what the database schema should look like, mock up an interface (with the ever-popular Microsoft Excel, tool of armchair developers everywhere), and otherwise paint the developer into corners that should never have been so much as stepped near.

I’m not sure what’s so difficult to grasp about the idea that software will better serve the user if the user is actually involved in that software’s creation. And not just sitting them down in front of a feature list and asking for a signature, either. For most users, “involvement” means showing them something that will make sense to them. This is rarely a bunch of buzzwords and maybe a bulleted list or two. I believe it’s a lot more useful to get a developer who knows what he’s doing in the room with a few users and start drawing data models, UI mockups, and walking through a storyboard of how a user might interact with the app. This has meaning to the users, allows them to offer some kind of meaningful feedback on the creation of the app. Nothing is more frustrating for me as a developer than to be delivered a spec that’s already been signed off on by the user, but addresses the “how” and not the “why” of the application.

Often, taking a few steps back and asking the user what they’re really looking to accomplish will yield a much more simple and elegant solution.

Captains of our own destiny

At a previous employer (one of the nation’s largest cable MSOs), I got the opportunity to write a customized trouble ticketing system for their commercial services business unit.

My boss had essentially given me carte blanche on the project. She was great to me at the time, stepping back and letting me do my thing, getting me support when I needed it. It was basically a one man show. I got to do the design, programming, project management, quality assurance. All of it. It was my dream project.

I had a previous background in technical support, which allowed me to get a few things worked out early on – it was easy to put myself in the user’s shoes when only 3 or 4 years ago, I would have been the user. The project would be written entirely from scratch, and completely customized for the way our support organization really worked. It would have a GUI front-end written in Perl/Tk, with a MySQL back-end, along with some server-based processes that handled user provisioning and e-mailed reporting. This way the UI would be highly responsive and not browser-dependent (this was before the explosion of well-tested cross-browser AJAX libraries we now have).

I spent months slaving away over every little thing – sweating all the details. I had done a bunch of Perl coding before, but never Perl/Tk, and many of its OO-style interfaces took some time to get used to. I would regularly go over ideas as I had them with the users, sometimes pull some of the more skilled users into my office to show them the current build of the app and solicit feedback. As my passion and excitement about this app grew, so did the users’. Granted, it was just a trouble ticketing app. It wasn’t going to save the world. Still, it would be a far cry better than the tool they were regularly being threatened with: BMC Remedy.

Know thy enemy

Almost anyone who’s worked in a support/help desk environment will recognize the BMC Remedy Action Request System. It’s an enterprise-class ticketing system.

Enterprise-class (adj.)

An IT industry term. Most typically refers to software that is too expensive, complex, and bloated to be used by anyone who actually exists on the planet Earth. Traditionally marked by at least three of the following:

• X-compliance, where X is whatever acronym or buzzword is currently most trendy in the relevant industry
• Slower than free or cheap alternatives for companies with fewer than X employees/customers, where X is the number of employees/customers in your organization +1
• Per-seat or per-user license fees
• Absence of any pricing information on the product web site
• A sales staff that is either highly motivated or just high on amphetamines, one (or many) of whom will call you upon your indication of interest in downloading a demonstration copy of the software
• Printed documentation available (for an additional fee)
• Developer API available (for an additional fee)
• An accompanying technical training and certification program (for an additional fee)
• Inexplicable absence of killer features that have been present in competing, but smaller, applications for years. In some cases, similar features are available via add-on (for an additional fee)
• A vast ecosystem of consulting firms professing to offer “solutions” based on the software (for a hefty fee)

To be fair, the intent of so-called enterprise-class software is to create software that will (supposedly) scale much better than other software. In practice, this doesn’t always work as advertised, and the chief benefit of choosing an enterprise-class solution is usually that there is someone else to blame when something goes wrong. Never underestimate the value proposition of a willing patsy. Enterprise-class software gets sold from the top down to people more worried about protecting their paycheck than getting the job done, not recommended from the bottom up by someone who will actually use it.

Now, with that out of the way, BMC Remedy fits more of the above bullet points than not, and therefore qualifies as enterprise-class. Solutions developed using Remedy are generally anathema to support representatives, unless those reps have never been exposed to something better. So it was easy to define my “enemy.” I knew precisely what we didn’t want in a ticketing system, so I set that in the back of my mind as I worked. In fact, an early concept for the name of the application was Malady, in homage to this.

Minimize pain for the greatest number of users

This seems redundant, right? I mean, I already mentioned that I had previously been showing this application to users as we went along. It stands to reason that this part takes care of itself. Well, kind of.

What I mean by “users” here is a little different. While we had a number of employees actually using this ticketing system, what group contained the greatest number of users? Our customer base. In the end, the customers were the users of the ticketing system, and our support reps were essentially their “interface” to it. I made a choice very early on during the development of PALADIN (this is what I ended up calling it – it was an acronym, but a pretty contrived one) to do my best to minimize pain to the customer, first and foremost, and then the support rep, then supervisors, managers, and other people needing reports, and lastly, the person who would be responsible for writing reporting code, me.

Without going into every little example here, one hotly debated choice I made (and which we eventually later compromised on) was to only support two “statuses” for tickets. Open and closed. The answer to which status the ticket should be in was simple. Was the issue resolved from the customer’s perspective? If not, the ticket was open. If so, it was closed.

In previous ticketing systems I had seen and used, any other status would end up becoming a graveyard for tickets. So many tickets in so many ticketing systems, damned to the limbo of “Pending.” Likewise, in deference to the number crunchers that live for reports, EVERYTHING ends up in a dropdown list in most ticketing systems, for nice, neat, classification on someone’s pie charts. This is complete and utter nonsense. There is no reason for the rep (and by extension, the customer) to suffer to suit the needs of a few who want to see pretty reports. Even if it meant more work in mining data from the DB on my part to extract things of use to them, I would rather deal with that than have every single call hindered by a cumbersome UI with 20 different dropdown menus when much of the information collected was irrelevant or redundant.

Details on why a ticket was left open or what work was being done was to be typed in a details field in plain English. At one point, to help increase ticket accuracy, I actually became a trainer, too. We did some custom training on example bad ticket data and good ticket data. I would wager that a few employees still remember “Pally the Paladin,” my attempt to try and add a little bit of levity to a training session that could otherwise come off a little too serious for my taste.

The ticket database was centered around customers and users (the reps on the phone), first and foremost, not tickets. Customer data would be entered once and then kept up to date. Tickets for each customer would simply be attached. This seems really simple, but many ticketing systems work on a “queue” basis, where there’s just a series of tickets, and often end up having customer ticket history lost in the shuffle, or forcing a needless re-entry of all relevant customer data every time the customer called in. This type of tedium is enough to drive any person mad, so I eliminated it. Turnover is bad enough in call centers already without making the job harder than it needs to be.

A useful side effect of these little niceties that we added with user feedback was that it decreased the tension a rep would convey on the phone to a customer, too, so even changes benefiting the second largest user base would indirectly benefit the customers.

Going live

In the initial release of PALADIN, on October 16th, 2003, only customer name, address, and tickets were saved. It had an integrated message center at the time, in lieu of e-mail, since some users didn’t have e-mail accounts on our corporate server. From then until 3 years later, in October of 2006, when I left the company, continued focus groups were held with users, and numerous enhancements were made, including a full refactoring of the database schema.

I integrated separation of customer contacts from the service location (“site”), tracking of multiple contacts per site, equipment addresses and serial numbers located at each site, and one-click real-time polling of current modem signal quality. I added complex ticket relationship hierarchies and parent/child relationships. In the end, while I made features prove themselves before they made their way in, the users got a lot of say in what the end result looked like, and as such, they were heavily invested in the application. They were the application’s biggest defenders.

A challenger appears

Initially, we ran PALADIN essentially “under the radar.” Nobody very much cared what program we used to track our tickets. We were the best-performing business unit in the company, and that kept folks off our backs. Eventually, though, as we drew the attention of more and more people from Corporate, the visitors would ask to see our toolset, and the question “What is PALADIN?” came up, since much of the company assumed we used Remedy, the corporate standard for residential support (remember, it’s enterprise-class!). One of the most satisfying moments in my career was when a visitor asked an employee that question, and the following exchange occurred…

“What is PALADIN?”
“It’s like Remedy, but it doesn’t suck.”
“Oh, okay!”

Of course, eventually, it began to irritate the Powers-That-Be that we had this little “homebrew” (you must read that with a derisive tone to truly understand the view the company had of most non-corporate development work) application which was consistently being used and praised throughout our little world. And then, the cries for us to conform came. To his eternal credit, my boss at that time (Hi, Ron!) responded “PALADIN is getting the job done for us, and does things that the current Remedy implementation doesn’t. As soon as you show us something that works as well for our needs, we’ll switch.”

There were many meetings around that time, and I visited the corp offices on several occasions to work on some interoperability projects. Nothing ever came of them, unfortunately. It was like we spoke two different languages. Every time we started talking about database structure, for instance, they would approach it from a data warehouse/reporting viewpoint, and I kept talking about ticketing needing to be transactional and user-focused.

On a fairly consistent basis, the concern, “What if Ernie gets hit by a beer truck?” would be brought up. We would explain that the code was readable, and documented, and that because it’s all open-source, there’s not going to be a huge problem in having someone else pick it up. Of course, when a company says, “What if you get hit by a truck?” what they’re really saying is “We don’t trust that you did/will do the right thing, and don’t want you to screw over the company and its investors.” Listen, I understand. When you’re a publicly traded company you have be careful. Still, if you hire decent programmers, you don’t have anything to fear. When I left, I did hand off maintenance of the app, and was told that the guys who took it over have had no real problems doing so, and added some great features since then (Hi Ben and Brian!).

Reinventing the wheel

Before I ever left, I remember on several occasions hearing murmurs that they would be trying to re-implement PALADIN within the Remedy Action Request System framework to satisfy our requirements. I’ve been gone for about a year and a half, and they must have actually taken up that task during this time, because this week, they finally rolled out their Remedy-based re-implementation of PALADIN.

As I understand it, it has numerous and exotic bugs, suffers from lockups, and requires adherence to the ridiculously expensive (and restrictive) AR System licenses for all users actively logging calls. It benchmarks at approximately 1/20th the speed of the original application in performing common functions.

I don’t pretend to know the whole story, and I’m sure there could be good reasons for all of this, but in the end, the reward of User-Driven Development is this, which was sent to me by one of my application’s former users (with the names/logos removed, of course):

Sent: Thursday, March 27, 2008 8:10 PM Subject: Funeral for a friend

Not really sure if any of you had a chance to attend, but if not here’s some photos from the funeral.

This, my friends, is what you get for putting your users first, and letting them take part in the application design process. Your users will be your application’s biggest promoters.

Thanks for reading.